Skip to content

feat(scheduler): heartbeat-aware health check for scheduled sync - #9

Merged
lurrnet merged 9 commits into
mainfrom
feat/scheduler-heartbeat-healthcheck
Nov 10, 2026
Merged

lurrnet merged 9 commits into
mainfrom
feat/scheduler-heartbeat-healthcheck

Conversation

@lurrnet

@lurrnet lurrnet commented Nov 10, 2026

Copy link
Copy Markdown
Owner

Why

The HTTP-based image HEALTHCHECK is only suitable for the web process. PR #8 disabled it for scheduler as a short-term correction. This follow-up adds a scheduler-specific, non-HTTP heartbeat/deadline check.

Implementation

  • Atomically record non-sensitive scheduler state in data/scheduler_heartbeat.json (0600): PID, start/heartbeat timestamps, sleep vs running, next scheduled run, previous completed run start/end and return code.
  • Wake from normal daily sleep every 30 seconds to pulse; pulse during long interval-sync subprocess waits.
  • Check process existence, heartbeat freshness (180 sec), overdue scheduled run (15-minute grace), and max running duration (4 hours).
  • Preserve last completed result while next run is in progress; failed remote sync is reported, not conflated with scheduler liveness.
  • Do not overwrite persistent heartbeat for --once jobs sharing the same mounted data directory.
  • Replace PR fix(compose): disable inherited healthcheck for scheduler #8's scheduler.healthcheck.disable: true with a Python-based Compose healthcheck. Keep the web image's HTTP healthcheck unchanged.
  • Add unit tests and operational README instructions.

Deployment and caution

This is a separate follow-up to merged PR #8, not a hotfix to the currently running OCI container. Rebuild the image and recreate services only after CI and review. Docker's unhealthy status does not automatically restart containers; configure monitoring separately.

Validation

GitHub CI: Python tests, frontend build, Docker smoke, history secret scan.

@lurrnet
lurrnet merged commit 63daab1 into main Nov 10, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant